I want to take you to a journey into the future, where after reading this article, a movie like Matrix won’t seems like a such a fantasy, but something realistic that might be achievable not far in the future and by just wearing mixed reality glasses. We will take this journey step by step. But keep in mind, there is so much to write about here, and I did my best to keep it relatively short. I also keep in mind that I don’t have the required knowledge to give an in-depth analysis of each aspect of the technologies required to make it happen, but I’ll do my best with what I do know. I hope this inspires others to learn more about mixed reality.
It’s also important to understand that the best experiences are those who blend with the environment, not replacing it with another virtual alternative. For this we have VR. However, there are ways that you can use different technologies to deliver unique experiences that are only possible when you have a view of the world, therefore possible in mixed reality applications. You’ll soon see what I’ve talking about.
Now, let’s the journey begin.
Present Times
Wouldn’t it be great to have a technology that can not just identify an object, but create a 3D replication of it, so you can manipulate it and mask the original physical object from any angle so now the virtual object replaced the original one?
My vision of a future without restrictions (I had to, so I can free my mind), AR Cloud is used to fuel this technology, storing not just spatial data (3D reconstruction of the real world space) but also imaging data. That imaging data is needed in order to be able to provide authentic reconstructing of content that needed to be masked out.
For example, take a look at this image and I will continue to explain. Please, forgive my bad illustration skills, I just try to make a point.:
The process of being able to authentically replace real objects (even an entire scene) with its virtual counterpart requires several steps: Now, obviously the physical object still exist, LOL, we are not doing any real magic here and we are not in a science fiction movie. So tihs technology will be limited to certain type of applications, we don’t want anyone to get hurt by walking over a table that doesn’t exist in the user’s view but it does in the real world 🙂
Context-aware 3D scene recognition – The algorithm needs to understand that there is a cup on the table and that the table isn’t part of the cup. Being able to distinguish between many existing objects in the scene. this should allow us to create a context-aware layers scene representation.
360° scene scan combined with AR Cloud 3D and Imagery data – A scene scan is needed in order to be able to construct an accurate 3D model of physical objects. Moving around the scene allows us to sample texture, analyze light and use algorithms that allow us to reconstruct the model in an accurate way. So textures like a book cover will be maintained, but the plain white color of the glass can be replaced by a solid color and shadow and lighting can be applied later to the 3D model. Without it, if you rotate that cup to the other side, the other side would have shadows and the front wouldn’t, opposite of how it suppose to look like based on the current scene lighting.
AR Cloud imagery data is good to help maintain a historical state of the structure, so machine learning algorithms can help us analyze the scene better. By this, I mean better scene recognition capability and the ability to feel in texture data where the object is being masked.
The Cloud imagery data is needed to make this physical object masking functionality faster. For example, how does the masking algorithm know what’s under the cup? It needs to know to be able to put content there (e.g. texture) to fill up that gap. One way to do it is to use advanced AI masking algorithms that can “guess” what should be underneath that cup. Using AR Cloud, that scene is reconstructed online, with not just point cloud data, but actual images. This way, the 3D scene reconstruction algorithm knows what’s below that cup, because it samples it many times in the past and based on its scene analysis algorithms, it can bring up that data that represent that areas when the cup wasn’t laid on top of the table. Of course, other objects could have been there, but that’s what that algorithm should do, keep a history and try to match that area with previous scans, the area around it and also asses the lighting so that the area can be represented faithfully and blend in well with the current lighting of the environment,
Now imagine taking this technology to a public outdoor area. This, of course, can be done as well, but obviously, the AR Cloud needs to be seeded with data from many users to be able to make faithful masking and reconstruction of physical objects. This is especially important for an evern changing physical landscape.
This will definitely require a lot of computer processing power if this should work fast. Somewhere in the future, we will have it working in real time.
Imagining the Future
The first iterations of this technology will probably require a lot of manual interventions. For example, moving around the apartment to provide the system with a full scan. Picking up the cup so it can asses how the surface looks underneath. However, these type of system should have deep learning capabilities, to make it work seamlessly, efficiently and more authentically.
This technology will literally transform our reality. You might have an app where you can literally pick up buildings in your hand and throw them out in the sky and see what’s on the other side. Hell, just smash them all and see how the entire blocks look without and buildings. You can take one building and place it instead of another one.
This can be used for architectural design uses. You can “delete” a physical building and replace it with a new model right in front of you. You will be able to “clean” your room in one click. You have apps that will allow you to browse the history of the world in front of you. Let me be more precise. Imagine being outside wearing your Mixed Reality glasses. You can just use a slider to slide back and see how that place looked at that same house 1 year back. We are like walking 3D cameras, walking around, archving out world. Imagine if this technology existed and we could go back 1000 years back. We can’t, but those who leave in the future will be able to do so.
Let me go even further. Imaging many people using MR glasses. Law enforcement can go out and analyze the entire city (or even cities) at a certain point in time. Hey, let’s go even further into the future. You can watch the entire scene in front of you like a movie. Go back and forth in time viewing all the information that was seeded to a central location by users at the time they were present and seeded the content from the scene.
Users will be able to even remove people, buildings, pets, and filter out content with just a few clicks. You can see how the world looked yesterday without people, or maybe just see the people without the world. You can have all that with 3D spatial sound or not.
Imagine a friend coming and telling you about a car chase that happened yesterday, or how amazing fireworks in the sky, you can go to that place, ho back in time and see it for yourself (if that scene was indeed seeded). In fact, you don’t need to go anywhere to see it, you can just browse through it from the comfort of your own chair in virtual reality.
Now, being able to do all that I describe, means that theoretically, you can replace the entire world with virtual content, or paint it with one. You can recreate new worlds with a click. This is a world that can turn completely virtual. Making you think that the Matrix (movie) reality can happen with just having MR glasses in front of your eyes. Crazy thought when you come to think of it. However, when you read this article, it doesn’t seem a fantasy at all, right?
I can go on an on if I just had more time today. I will probably extend on it in future articles. It’s just walking alongside technology and follow its steps, feeling up things using my own imagination, part of it based on human needs, fantasies, wishes, etc.
Summary
All that discussion started as a small topic, but as you can see, this can get really out of control pretty quickly. It’s important to understand that many of those topics are way above my knowledge. I just try to put on writing what’s in my head. There might be more efficient and better ways to explain it or to execute it.
All I want is to open peoples minds, share my enthusiasm of how my mind see the future of mixed reality. When I landed back on Earth, I saw how early we are in the process of getting there. Still, it might happen sooner than we think.
Of course, there are many things related to privacy that I didn’t even mention here. I wanted to let my mind go, and I can’t do it if I put restrictions on my thoughts. when I go on a trip, I let myself go of any obstacles, this is the only way I can “travel” to the future.
Anyways, I hope that this topic will open up a fruitful discussion about the future of the mixed reality technology. There are so many amazing uses of MR that can improve the lives of every one of us. I’m sure there will be many obstacles along the way, but eventually, I think that will get there, with some restrictions.
Excuse me for grammar mistakes that I make. I had little time. I also wanted to add some amazing images to illustrate things, but again, due to lack of time I wasn’t able to do it. If you like this article, please share it. Thank you.
Baner image credit: Gerd Leonhad. Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0).